Cue Coupling

I. VISION MODULES AND CUE COMBINATION To what extent is the visual system built by combining modular components? This issue was discussed in the basic architecture section and viewpoints differed between advocates of separate pathways encoding different feature dimensions [9], [17] and other who argued against separation because most cells are sensitive to all dimensions [16] and, premature separation, raises concerns about how different pathways can be combined in order to yield a unified percept. At the functional/behavioral level, psychophysicists have studied how humans combine different visual cues – such as shading, texture, binocular stereo, structure from motion – to get depth. Marr [20] invoked the principle of modular design and proposed that different cues could be processed by semi-independent modules and then combined into representations – e.g., Marr’s 2 1/2 D sketch combined different cues for depth into a single representation. Computer vision researchers have also tended to study these cues in isolation and developed algorithms from them individually. We argue that although there is evidence that cues can act independently (and deficit studies show that damage to localized brain regions is able to knock out some visual cues while leaving others unimpaired). Nevertheless, evidence suggests that the modules must be capable of tightly interacting in certain situations. In particular, there is strong evidence that high-level recognition affects the estimation of three-dimensional shape (e.g., a rigidly rotating inverted face mask is perceived as non-rigidly deforming face, while most rigidly rotating objects are perceived to be rigid). It is also clear that, despite the partial success of the models described for segmentation in section (??) (and their more sophisticated descendants), the ability to perform segmentation relies partially on context and the ability to do object detection, hence involving some top-down processing. This section gives an overview of visual cues and the strategies for combining them. Clark and Yuille [4] argued that cue coupling should be formulated in terms of Bayesian probability theory so that the uncertainties of the cues could be taken into account and their statistical dependencies made explicit (previous authors, e.g., Marr [20], had not specified the details of how cues should be combined). Clark and Yuille [4], see also [24], divided cues coupling into two types: (i) ’weak coupling’ which corresponded to combining independent cues and which often reduced to combining cues by weighted averaging, and (ii) ’strong coupling’ which required more sophisticated methods such as model selection, or ‘competitive priors’, to model phenomena where small changes of one cue can dramatically change the percept. Since then many studies have shown that humans often couple cues weakly in an optimal sense – i.e., that they weight the cues based on their reliability. To understand this it helps to consider how the image is formed from the structure of the real world. We can think of this in terms of causes. Different factors in the scene combine to cause the image. These combinations can be complicated and are rarely independent – as they would need to be for weak coupling to be appropriate. In addition, some visual cues are only valid under special conditions. For example, classic shape from texture methods assume that there are texture elements of roughly similar sizes on a surface in the scene. This assumption can be relaxed but nevertheless there are many places in images where shape from texture cues simply do not exist. Similarly, classic shape from shading models apply only to Lambertian shading models with known albedo and a single light source. It is helpful to think of cue combination in terms of graphical models as we illustrate in figure (6). We stress that these diagrams are conceptual and ignore the details of the models – i.e. each node should